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Abstract 


Online delivery methods ojfer much promise for anytime, anywhere adult 
learning. As a method of outreach, and to provide support for just-in- 
time learning, teacher educators are increasingly deciding to design Web 
sites that are online, resource-based learning environments for teachers 
and preservice teachers. Automated evaluation tools and data collection 
methods can help such Web site designers develop and use online, resource- 
based learning environments to meet their goals of providing high qmlity 
learning opportunities for teachers. This article presents a three-pronged 
approach for evaluating issues of pedagogical design and user-centered 
functionality in online, resource-basedWeb sites. Checklist, Web site traffic 
analysis, and talk aloud protocols are each described and then illustrated 
through their application to a site aimed at educating teachers about 
technology integration. We discuss the relevance for developing effective 
learning environments when teacher educators apply this evaluation ap- 
proach to resource-based learning environments. 

T eacher educators recognize that online delivery methods ofFer 
much promise for anytime, anywhere adult learning. In addition 
to courses taught partially or entirely online, teacher educators 
are composing collections of resources in learning environments that are 
designed to support the professional development needs of teachers for 
informal learning or just-in-time needs (Dede, 2003; Partnership for 
21st Century Skills, 2003). Examples include Weh sites designed hy staff 
members of professional organizations (e.g.. National Science Teachers 
Association at http://www.nsta.org), through professional development 
projects or grants targeting materials to a specific audience (e.g.. Learning 
Point Associate’s “enGauge” at http://www.ncrel.org/engauge/), or even 
hy individuals with a special interest (e.g., Kathy Schrock’s resources at 
http://schooI.discovery.com/schrockguide/). 

For teacher educators in the role of Weh site designer' seeking to 
create and use such online, resource-hased learning environments, find- 
ing the answers to the following questions is essential: What Weh site 
characteristics are associated with a quality online learning environment? 
How can the features and resources of a Weh site he designed to support 
informal, just-in-time learning? Which aspects of the site most engage 
users? Which users are drawn into the site? We first sought to answer these 
questions when evaluating the quality and usefulness of eTech (pseud- 
onym^), an online, resource-hased learning environment for teachers 
organized around issues of technology integration. The environment was 
produced as part of a federally funded Preparing Tomorrow’s Teachers to 
Use Technology (PT3) project on which the three authors worked. We 
developed and piloted a multi-method evaluation approach outlined in 


this article as a resource to help teacher educators evaluate the quality 
of the Weh sites they draw on as resources, or serve as a resource when 
designing such sites. 

This article first defines and describes resource-based learning. Next, a 
multi-method approach is described for evaluating online, resource-based 
learning environments. These methods are illustrated through a review 
of the eTech Web site. We conclude with a discussion of the possibili- 
ties for teacher educators when applying these methods to improve the 
design and utilization of online, resource-based learning environments 
for teacher education. 

Review of the Literature 

Online Resource-based Learning Environments 

Unlike online course environments, such as WebCT or Blackboard, 
sites designed as learning environments that are without a particu- 
lar instructor, a designated term or assignments, or any particular 
class list, offer a distinct type of learning referred to in education as 
resource-based learning. These environments can be an important 
supplement to teacher education programs (Davis & White, 2001). 
Resource-based learning operates on the premise that learning can 
be facilitated with access to information organized around a specific 
domain that learners can actively explore (Davis & White, 2001). 
Online, resource-based learning environments are designed for a 
particular audience and educational purpose. In teacher education, 
for example, many school districts, higher education outreach efforts, 
and non-profit organizations are looking to online, resource-based 
sites to support pre and inservice teachers who are expanding their 
pedagogical repertoire. Features of such sites may include hypertext 
links, which enable users to seek and explore information on-demand, 
(i.e., content area standards or lesson plans), multimedia presenta- 
tion (i.e., video clips, pictures, spreadsheets, graphs, and sounds), 
search capabilities, communication tools (i.e., threaded discussion 
forums or chats), customizable interfaces, and other resources that 
can potentially support users’ active exploration within a particular 
domain (Davis & White, 2001; Cameron & Spaeth, 2000; Jonassen, 
2000) . While much has been written about the Web as a resource for 
learning in higher education, these pieces focus mainly on models or 
software packages for distance education (Chang, Lin, Hao, Suarez, 
& St. Lawrence, 1997; Fryer, 1997) or the necessary infrastructure 
and technical support needed to deploy them, the design of the user 
interface for online course environments (Kaye, 1996; Descy, 1997), 
or promise of generating tuition-paying students (Khan, 1997). 


' We use the term “Web site designer” to refer to individuals, such as the staff of a project or institution, who seek to have educational materials made 
available to a targeted audience, and in many cases, evaluate and improve upon their use. We distinguish “Web site designer” from the “Web master.” 
Although Web masters may also participate in Web site design, they input and adjust site content, install Web site analysis software, and help troubleshoot. 
^ While the evaluation of eTech was the catalyst for the methods we describe here, in this article it serves mainly as an example of their application. 
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Purposes of Evaluating Online, 
Resource-based Learning Environments 

The challenge for teacher educators that invest in resource-hased learn- 
ing sites is to assess the site’s potential and effectiveness as a learning 
environment (Barha & Clark, 2002; Swann & Einstein, 2000). If the 
more traditional process and outcome measures that are used for formal 
online course environments do not apply, then teacher educators who are 
designing or evaluating resource-hased Weh sites must find alternative 
methods to evaluate such aspects as the quality of design, user characteris- 
tics, preferences and behaviors (Trochim, 1996). For example, without an 
authenticated login, attendance and rate of participation or other process 
indicators are not collected; without a term limit or assignments and tests 
teacher educators cannot look to the completion rate or quality of assign- 
ments or scores as indicators of the users’ learning. While the literature 
on evaluating educational Weh sites is extensive, the vast majority of this 
has been summative in nature, delivering criteria that instructors and 
students can use to decide whether or not to use or ignore a Web site as 
a source of information. Much of this literature provides a set of criteria, 
including scope, presentation, reliability, and match to curriculum to help 
people compare sites for appropriateness in a course (Branch, Dohun, & 
Koenecke, 1999; Wilkinson, Bennett, & Oliver, 1997; Schrock, 1998). 
Much less has been written on formative approaches to evaluating educa- 
tional Web sites that provide information to designers in their planning 
and development of a Web site. These approaches can be divided between 
situations where the feedback from the intended users is readily available, 
such as where an instructor develops a Web site for his or her own course, 
and situations where a teacher educator seeks to provide instruction via 
an online, resource-based learning environment but will not receive 
substantial direct feedback from users about it (Hughes, 2001; Rieger 
& Sturgill, 1999; Weston & Barker, 2001). The evaluation approach we 
describe here aims to inform the latter situation. 

Multi-method Approach to Formative 
Evaluation of Learning Environments 

The evaluation approach we present here is formative and draws heav- 
ily on a utilization-focused evaluation model (Patton, 1997) in that it 
provides information to teacher educators interested in improving Web 
sites that they are designing or reviewing for use. We do not address 
the processes implied in utilization-focused evaluation for tailoring the 
evaluation to the audience’s needs, but rather present a set of evaluation 
methods sufficiently flexible to be adapted to different contexts in which 
resource-based Web sites are constructed. 

Drawing on research on the key dimensions of effective learning en- 
vironments as summarized by Bransford, Brown, and Cocking (2000), 
we used these dimensions as a checklist of features desired in an online 
learning environment and applied the checklist of criteria to the eTech site. 
This method was complemented by two user-centered research methods: 
Web site traffic analysis, which draws on data about the length, frequency, 
and repetition of site visits to infer basic patterns about Web site use and 
users, and think aloud exercises, which supplement the lack of contextual- 
ized information provided by traffic analysis and give examples of how a 
user may perform and experience tasks in the Web-based environment. 
These methods were used in a parallel manner, although findings from 
each method can inform the use of other methods (e.g., pedagogical design 
shapes the tasks users are asked to do in think aloud exercises). 

Evaluating Online Pedagogical Design: 

Key Dimensions Checklist 

Online, resource-based learning environments are most likely to be ef- 
fective when they are designed and utilized in ways consistent with the 
research on how people learn. Bransford, Brown, and Cocking (2000) 


base the following key dimensions of effective learning environments 
upon current educational and cognitive neuroscience research. 

Knowledge-centered environments focus on developing knowledge 
of the field or discipline and of strategies to develop expertise. They 
incorporate the language, artifacts, and essential principles on which 
learning in the discipline is based and model how experts work with those 
principles to gain increased understanding. In a Web-based environment, 
this might translate into digital, discipline-specific materials that are up 
to date, authentic, and accessible to the target audience. 

Community-centered environments evolve a community of practice 
among like-minded professionals where members develop common goals 
and work toward achieving them. The community-centered environment 
provides opportunities for active participation and the development 
of a shared knowledge base. In a Web-based environment, this could 
be expressed as Internet-based conferencing environments that assist 
communication over distances, or a repository of artifacts of members’ 
practice. 

Assessment-centered environments support learners’ testing of ideas 
by promoting ongoing reflection and feedback on practice. Such environ- 
ments enable opportunities for metacognition in a sustained, coherent 
context. Effective Web sites might include Internet-based networking to 
promote ongoing reflection and feedback on field experiences in a manner 
that is sustainable, wide reaching, and affordable. 

Learner-centered environments focus on learners, building on their 
strengths, interests, and needs. They take individual learning styles and 
prior knowledge into account. One of the most promising and under- 
appreciated qualities of new information technologies is their interactive 
capacity. Effective Web sites might allow users to customize the site ac- 
cording to their preferences and expertise. 

The four dimensions or environments outlined here can be visualized 
as interdependent and overlapping spheres; therefore, we might expect 
all four dimensions to be present and functioning in a quality learning 
environment. Referencing research that suggests Web-based technology 
capabilities can be used as tools to scaffold, motivate, and enhance think- 
ing (Chan, Burtis & Bereiter 1997; Gordin & Pea, 1995; Jonassen, 2000) 
and referencing Bransford et al’s conceptual framework about effective 
learning environments, we developed a checklist of the desirable features 
one would want to see in an effective Web-based learning environment 
(see Table 1). The purpose of applying a checklist is to guide a focused 
content analysis of the online, resource-based learning environment be- 
ing evaluated in order to identify its relative strengths and weaknesses. 
Identifying which items are absent or present is a helpful first step in 
taking a critical look at how present items support the site’s educational 
purpose and intended audience. This also helps to clarify utilization and 
development strategies, such as which contexts are most promising for 
improving the resource-based learning environment. Thus, the purpose 
for the checklist’s use is not about the number of checks that amass, but 
rather how this systematic review, against established criteria, highlights 
the support for learning that the Web site provides or does not provide. 

The main advantage for teacher educators who use such checklist cri- 
teria to guide data collection and analysis is that they link Web site design 
to a research based on quality instructional practice and they prompt the 
consideration of facets of the learner’s experience not otherwise appar- 
ent. Checklists can facilitate a more systematic and efficient approach 
to collecting and categorizing information, and help teacher educators 
more readily identify patterns in the data than may be possible with a 
grounded approach to inquiry. 

While the framework we have outlined above is helpful for evaluat- 
ing the pedagogical design of a Web-based learning environment and 
its potential for supporting learner’s growth, other methods and tools 
are needed to complete a user-centered analysis of the environment. A 
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Table 1. Key Dimensions of 

Online Learning Environments Checklist 


Possible Features of a Knowledge-centered, Online Learning Environment 

• Hyper-linked resource area (e.g., links to content area standards, current 
research on Methods and Learning Theory, etc.) 

• Digital content-area, curriculum or resources focused around specific 
issues or themes that is up-to-date, authentic, accessible and generative 

• Electronic networking forums to support knowledge sharing (e.g., Web- 
based libraries of: videos of teachers and accompanying descriptions 
exchanged, reviewed, annotated, and linked to additional resources; lesson 
plans; journal entries; student work) 

• Interactive design tools (e.g., curriculum design tools) with electronic 
prompts to scaffold thinking 

• Electronic notebooks encourage reflection in design process 

• Interactive survey instruments encourage reflection on workplace practices 

Possible Features of a Community-centered, Online Learning Environment 

• Internet-based conferencing environments that assist communication over 
distances 

• Synchronous mechanisms (e.g., via "chat" rooms, multi-user virtual 
environments or asynchronous e-mail, bulletin boards, threaded discussion 
forums) 

• Asynchronous mechanisms (e.g., e-mail, bulletin boards, etc.) that allow 
anytime-anywhere communication and archiving of discussions to assist 
the development of a group understanding over time 

• Coherent human infrastructure (i.e., human facilitators who moderate online 
conferencing environments) 

• Repository of artifacts of professional practice (e.g., sites aimed at 
teacher professional development might have examples of students' work, 
collaboratively designed lesson plans, electronic records of previous 
online discussions, URLs to relevant information, etc.) that can be quickly 
retrieved, distributed, organized, and stored within the online community 

Possible Features of an Assessment-centered, Online Learning Envirenment 

• Tools for or tools that encourage self-assessment, reflection, metacognition, 
feedback such as pop-ups ortext boxes to write selected information (e.g., 
an electronic notepad for recording reflections, pop-up questions prompting 
self-assessment, electronic feedback forms, etc.) 

• Tools for or tools that encourage peer exchange, constructive critique, 
networking, such as collaborative design tools (e.g., shared online spaces 
for posting and editing group work, etc.) 

Possible Features of a Learner-centered, Online Learning Envirenment 

• Data in graphical, text-based, and multimedia forms, acknowledging diverse 
learning styles 

• Searchable online database through which users can search the site by a 
topic of interest to support professionals in important aspects of their daily 
work and let them easily and quickly find what is of interest to them 

• Customization according to user's preferences and knowledge (i.e., through 
Web-based tools such as search engines, databases, conversation tools, 
simulations, visualizations, and learning environments to give users the 
ability to initiate discussion strands, import objects, attach files, store work, 
download software, post notes and published work, or add sound and video 
to the online workspace) 


user-centered analysis that considers user characteristics, preferences, and 
behaviors will assist Web site designers when they assess the success of 
the implementation of the above design elements. 

Evaluating Learners' Use: 

Web site Traffic Analysis 

Web site traffic analysis is one method that can help designers determine 
demographic information about site users (from where users come to the 
site and what learning resources they most often use while there). While 
this software-driven approach provides a cost-effective and comprehensive 
means of data collection, it supplies little context for the users’ activities 
at the Web site. 

One only has to do a search on the Web for “Web site traffic analysis 
software” to find the variety of commercial packages available to analyze 
how and when visitors are accessing a Web site. While a complete listing 
or review of such software is beyond the scope of this article, we include 
here some name brands: LiveStats (version 6.2), WebTrends (version 
5.0), 123LogAnalyzer (version 2.5), eXTReMe Tracker (version 1.0), 
and HitBox (version 1.0)^. There are also many Web sites that describe 
how such software packages analyze the statistics that are collected in a 
database on Web site servers (e.g., Itoh, 2001), and compare and contrast 
the features of such packages (e.g., Aviram, 1998). The programs differ 
as to the type and extent of reports they create based on these statistics, 
but nearly all such software generates similar basic measures, and is de- 
signed to provide information about users’ activity to Web site designers 
or server managers. 

Most of the reference information on these tools and their use illus- 
trates their application to commercially oriented Web sites (e.g., Lawson, 
Howard, Kennedy, & Pritchard, 2002; Morris, 1999), to help businesses 
see what products or information is most interesting to customers. Yet, by 
substituting learners for customers, and learning resources for products, 
designers of educational sites can generate insights about how this type of 
software can assist them. Through automatically generated and continu- 
ously collected data, such as that described in more detail below, designers 
of educational sites can gauge to some degree the “reach” of the site. They 
can gauge whether actual user profiles match the intended learner audience 
and identify which learning resources seem most appealing. 

Designers can determine whether the site is capturing a return audi- 
ence by reviewing reports on visitor demographics. While a high number 
of unique visitors might indicate that the site reaches a large audience, 
designers of an educational resource very likely expect people to return. 
Calculating the ratio of onetime visitors to “more-than-one-time” visitors 
can help designers infer whether the site was compelling or useful enough 
to convince learners to return. 

Data on the average length of visits versus the median length of visits 
can inform site designers if some users linger over the site. The median 
represents the mid-point of the data, dividing all users into two equal 
halves. That the average time spent by users is higher than the median 
suggests that some users are spending far longer than others at the site, 
and they are perhaps poring over resources. If the average and median 
time spent by users at the site are similar, one might infer that learners 
are uniform in their time spent at the site; of course, the length of time 
for either measure is also helpful information. 

Reports on when those longer visits occur might help site designers 
make some educated guesses about who is using their site. For example, 
one might expect that the peak traffic hours for a teacher resource site 
would be after school during the week. If the traffic analysis report suggests 


^ For further information, see LiveStats: http://www.deepmetrix.com; WebTrends: http://www.webtrends.com; 123LogAnalyzer: http:// 
www.123loganalyzer.com; eXTReMe Tracker: http://www.extreme-dm.com/tracking/; HitBox: http://www.websidestory.com/products/web-analytics/ 
hitbox-professional/. Note: eXTReMe Tracker and Hitbox software companies offered no version numbers at the time we used their software; therefore, we 
annotated these software packages as their original versions (version 1.0). 
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otherwise, it should prompt the designer to form questions and collect 
further information about how the site is being used and by whom. 

Data regarding the site resources accessed allows the Web site designer 
to discern the most and least accessed pages, along with information on 
the pages where most visitors enter and exit. This is important feedback 
that might suggest which of the site’s learning resources users perceived 
as most valuable. Such automatically generated data can give teacher 
educators /designers an idea of how evenly the resources on the site are 
being utilized, and if resources intended to be most valuable are recog- 
nized as such. This information can allow site designers to set priorities 
as to which content areas should be further developed, and which areas 
should be minimized. This information can also help the designers of a 
site decide how to optimize the architecture of the Web site based on 
where visitors are entering. 

Figuring a ratio of page views to unique visitors enables site designers 
to gauge how encouraged users are to venture into the site, follow links, 
and check out resources. This ratio also allows designers to infer some 
things about the visitors’ satisfaction with their visits. That is, if a visitor 
accessed many pages, especially within a particular topic area, she may 
have found resources helpful enough to look at more. 

Finally, data collected on referrers (i.e., links from which visitors come 
to the site) might suggest other entities with which teacher educators 
designing the Web site could seek partnership. Data on the keywords 
used in on-site searches or keyword searches, which led users to the site, 
helps designers learn what sorts of information users expect or hope to 
find there. If the things users are looking for are not the sorts of learning 
resources a site provides, designers might want to add, or link to alternate 
resources. Of course, data on keyword searches also helps define the meta 
tags to place on the site’s pages, so users who are looking for specific 
information land at the appropriate page. 

In summary, the Web traffic analysis is helpful in that it lays out the 
facts of a site’s use in a cost-efficient and comprehensive way, but it only 
goes so far to support the designer’s ability to make inferences about 
why site users did what they did. In order to learn why teachers move 
through the educational resources as they do, another research method 
is needed. In the following section, we discuss how we used a talk aloud 
protocol to understand user preferences and behaviors in our resource- 
based learning environment. 

Evaluating Learners' Understanding: 

Think Aloud Protocol 

Think aloud protocol, also known as talk alouds or verbal protocol 
analysis, is a technique for assessing cognition that involves asking a study 
participant to report his or her thoughts related to performance of a task. 
It is associated with a variety of contemporary research methods aimed 
at capturing thoughts as they occur, such as thought-listing or random- 
thought sampling (Clark, 1 997) . The technique typically involves asking 
participants to undertake a problem-solving task of moderate length while 
reporting their thoughts as they occur. The participant is urged not to 
think about what she is going to say before she says it, but rather focus 
on the task. To this end, the researcher usually sits behind the participant 
and only speaks to remind her to continue thinking aloud. Outcome 
measures include a transcript of the think aloud, which can be used in 
content analysis, and measures of task performance. The main advantage 
of the talk aloud method is that it can provide a detailed description of 
how subjects experience the task (in this case, using an educational Web 
site). The data provided by the think aloud is continuous over time and 
can be broken up into a number of discrete data points. 

Think aloud protocols have been used in educational research to 
evaluate instructional methods (Grave, Boshuizen, & Schmidt, 1996), 
assessment methods (Bartolo, Dockrell & hunt, 2001; Norris, 1990), 


and the use of instructional technology (Goldman, Zech, Biswas, Noser, 
et al, 1999; Mathison, Meyer, & Vargas, 1999). Indeed, Goldman and 
colleagues present a strong case that the assessment functions provided by 
computer traces are insufficient as assessment tools unless accompanied 
by process tracing methods such as think alouds. Ericsson and Simon 
(1993) suggest that there are three levels of thinking aloud affected by 
the type of instructions given to the subject. These include: (1) spontane- 
ous thinking aloud, such as what one might do naturally in the process 
of thinking through a problem; (2) intentional thinking aloud with the 
purpose of simply reporting thoughts but not explaining them to an 
audience; and (3) intentional thinking aloud with the purpose of com- 
municating one’s meaning to others. They recommend the second level 
as producing the most useful data that engenders the least reactivity to 
the method. Asking the subject to explain his or her thoughts (level 2) 
may cause significant changes in what is reported. In conducting think 
alouds with inservice teachers as opposed to preservice, the distinction 
between the second and third levels may become especially tenuous as 
teachers may be quite familiar with talking aloud in problem-solving as 
a way of communicating content in a classroom setting. Thus, it may be 
necessary to emphasize that the participant need not explain everything 
he or she says during a talk aloud. 

The main methodological issues regarding the think aloud protocol 
have focused on whether it truly captures cognitive processes and to what 
degree it is also measuring subjects’ reaction to the think aloud situa- 
tion. In general, think alouds have their highest validity when capturing 
conscious thoughts, which are easily verbalized, as they occur. The issue 
of reactivity calls attention to the importance of the nature of the task 
undertaken during a think aloud. Ericsson and Simon (1993) suggest that 
subjects should focus primarily on the task rather than the think aloud, in 
part so that the subject does not devote too much attention to thinking 
about what they are saying. In service of this aim, it is common procedure 
to allow subjects to go through a practice task so that they develop some 
comfort with thinking aloud. The think aloud should simulate actual tasks 
to be undertaken by the intended Web site audience and pose a moderate 
level of difficulty. Our own work in developing tasks for the eTech Web 
site evaluation described below suggests that the quality of data provided 
by the think aloud is related to task difficulty in an inverted U pattern. 
A task that is too easy requires little thinking and is performed quickly. 
A task that is too difficult constrains the cognitive resources necessary to 
perform the task and report thoughts at the same time. 

eTech Web Site Evaluation 

Data and Methods 

The multi-method framework outlined above was applied to the eTech 
Web site approximately two years after its startup and midway through 
the life of the eTech project. The eTech Project was designed to support 
preservice teachers during and after their teacher preparation program to 
use technology in support of instruction in their subject matter. Orga- 
nized into 1 1 content areas/teacher licensure programs at our university, 
the site included: vignettes illustrating added value uses of technology; 
hardware and software product information; links to external Web sites 
with teaching resources, lesson plans, and professional organizations; an 
online discussion area; and case-based interactive exercises for practicing 
instructional decision making. The site was introduced to preservice teach- 
ers during their required technology integration class, utilized throughout 
its duration, and intended to serve as a resource beyond graduation. The 
eTech project staff compiled the resources at the eTech Web site. 

The first two authors, acting as project coordinator and project director 
respectively, separately applied the checklist in Table 1 to the eTech Web 
site, noting what features were present or absent. Afterwards, they met 
with the third author to compare their checklists and discuss areas for 
improvement. Data on eTech Web site traffic was collected automatically 
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by WebTrends software (version. 5.0) (1995-2002). Quarterly summaries 
of Web site traffic from April 1, 2000 through April 1, 2002 were ana- 
lyzed. Specifically, the authors looked at the following: proportion of 
returning users out of total users, pages first accessed by users, length of 
time spent at the site overall, and most frequently accessed pages. The 
third author, acting as project evaluator, recruited two inservice and four 
preservice teachers to take part in a think aloud exercise in February and 
March 2002. These people were enrolled in teacher education courses at 
the major Midwestern university where the eTech project was based. The 
six teachers were asked to complete three short tasks while reporting their 
thoughts about the tasks to the third author. One task asked participants 
to find an idea for integrating technology into a fourth grade science 
lesson, while another task asked participants to find an idea to use in an 
eleventh grade geometry lesson. The other two tasks asked participants 
to find general advice and resources related to technology integration 
in the classroom. A brief interview following the exercise asked for the 
subject’s evaluation of the usefulness of the site and what features could 
be improved. Each think aloud was taped and transcribed for review by 
the project director and coordinator. 

Results of Checklist 

Through applying the key dimensions checklist to the eTech Website, we 
collected information on the range of technology capabilities present in 
the site, including their intended and potential uses, and categorized them 
along the knowledge-, learner-, assessment-, and community-centered 
dimensions. We were quickly able to identify the strengths and weaknesses 
of the site’s design and formulate questions about its content and archi- 
tecture for further investigation. For example, we determined that eTech 
incorporated some assessment-centered characteristics, but was weakest 
in this dimension of quality. Users were invited to send feedback about 
the site via a comments form on the homepage. In addition, users could 
link to online, case-based scenarios where they could practice making 
instructional decisions about technology. They could receive summative 
feedback on their performances, but the software did not provide forma- 
tive feedback, such as electronic prompts during performance. It also did 
not allow opportunities for self-assessment. 

Results of Web Site Traffic Analysis 

Analyzing WebTrends reports for the eTech site enabled us to make infer- 
ences about our users. For example, visitor demographics suggested that 
about one-third of eTech users were returning users, and that the average 
visit length was more than 10 minutes. eTech site resource numbers sug- 
gested that about a third of our visitors started at the homepage, from 
which the site branched out into K-12 subject areas. Other data suggested 
that most visitors moved from home to the beginning pages for these 
other 1 1 subject areas. These other 1 1 starting points were nearly evenly 
noted as entry pages. A resource page we host for a particular technol- 
ogy course at our university was both a top entry and exit page on our 
site. About half of our traffic had no referrer noted, and another large 
percentage came from our college or university Web site. Only a small 
portion of our audience arrived from a search engine. 

Results of Think Aloud Protocol 

With the two tasks that asked participants to search for a lesson plan 
idea, nearly all participants followed a similar path to accomplish the 
task, although some took less successful paths before this. They went 
to a subject area (e.g., science or math), clicked on resources, and then 
clicked on lesson plans. From there, some went to Web sites with which 
they were already acquainted (e.g., Marco Polo) and conducted searches 
from those Web sites. 

Interestingly, most participants found looking for general information 
on technology integration difficult. When asked to find a simple guide to 


technology integration on the eTech site, the six participants were evenly 
divided among three different search paths before undertaking the task. 
Two participants gave up after repeated, unsuccessful use of the on-site 
search engines; two participants found a summary of technology integra- 
tion principles by going to the site map; and two participants found a 
brief article on added value by going through an individual content area. 
For example, looking for a guide on how to integrate technology on the 
eTech site, a participant read the Web site aloud and commented on how 
useful they thought a possible guide might be. 

“Technology use provides added value.” That sounds 
kind of like what I’m looking for. “Appropriate to use 
in a lesson.” So “Technology use” I’m going to. No, 
this is just saying that it does — that technology use 
provides added value to teaching and learning. It’s 
more of j ust a statement, not whether or not the tech- 
nology is appropriate. I’m just reading these things. 

“Added value summary.” I mean I suppose this is kind 
of related. This is not really a guide to deciding. I guess 
a guide to me would be something that’s more like 
questions and like a checklist or something. 

The post-session interview revealed that some of the difficulties expe- 
rienced by users were due to incongruence between the users’ vocabulary 
and that used within the eTech Website. This applied especially to the 
terms “resources” and “tools.” One subject remarked during her talk aloud 
that the eTech Web site’s labels did not help her to understand what she 
might find there: 

To me, when I saw [the label] “Resources” I didn’t 
necessarily automatically go to ‘Oh yeah, a lesson 
plan,’ things like that. “Resources,” I think of maybe, 

I thought of what kind of technology could be used, 
or something, which is kind of included I guess. 

But, I guess “Resources” is a very general term - and 
which you may have very well intended since a lot 
of information is underneath it. 

Discussion of Results 

The multi-method approach to formative evaluation provided three differ- 
ent yet complementary views of the eTech Web site as an online, resource- 
based learning environment. Using the checklist criteria as an outline 
of what a learning environment based on good pedagogical principles 
should look like, we found the Web site weak on the assessment-centered 
dimension, especially with respect to formative assessment. The traffic 
analysis indicated that the intended audience (i.e., preservice teachers 
learning about instructional technology) was accessing the Web site, but 
users were not necessarily accessing the material in a linear fashion. Only a 
minority went through the homepage. The think aloud protocol provided 
a more nuanced view of Web site use and showed that such use is highly 
dependent on a clear understanding of the terms employed throughout 
the Web site. Since the study was conducted midway through the eTech 
project, the results were used to improve the Web site as a Web-based 
learning environment. For example, we made changes to the homepage 
and to the reference page for the technology course. We also reached out 
to people at our top referring sites, to help them become more aware of 
our resources and perhaps further build a partnership. 

One of the essential characteristics of an “internal” evaluation like this 
(one that allowed us to make responsive and timely changes to the Web 
site) is also one of its main limitations. An evaluation of a Web-based 
learning environment by its designers may bias the results obtained. While 
using checklist criteria to evaluate the pedagogical design of Web-based 
learning environments can be useful for generating formative feedback 
and assisting content analysis, the disadvantages of applying a checklist 
include the tendency toward oversimplification or inaccurate categoriza- 
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don depending on the evaluator’s training and knowledge. For example, 
the categorization of the eTech site’s degree of assessment-centeredness by 
the project director and coordinator may be overly simplified or biased 
toward intended uses rather than potential uses of Web site features, given 
that the reviewers were both experienced visitors to the site and familiar 
with eTech’s layout. Where data do not fit into predetermined categories, 
or categories are loosely defined, reviewers may adhere too closely to a 
perceived checklist and therefore miss relevant data or opportunities for 
further clarification of the instrument. 

To a certain extent, the use of complementary methods and data 
sources we describe here can help to overcome this bias. Web site traffic 
analysis is based on records of actual, rather than intended user behavior. 
Think alouds provide a close approximation of the user’s actual, rather 
than imagined experience within the learning environment. These latter 
two methods, in turn, are limited by providing only a description of a 
user’s experience and not a prescription of how a Web-based learning 
environment might be used successfully. For example, the results of the 
think aloud techniques are partially dependent on the particular skills 
and experiences that participants bring to the exercise (including experi- 
ences that poorly prepare them to use the eTech Web site). Even during 
the course of the think aloud, each participant tended to quickly learn 
the eTech site’s structure, such that the third and fourth tasks drew on 
knowledge they gained in earlier tasks. 

Conclusion 

The multi-method approach outlined above provides an alternative 
means for evaluating the pedagogical design and learners’ use of online, 
resource-based learning environments where more traditional process 
and outcome measures are unavailable. We have illustrated how these 
methods helped us to evaluate and develop a resource-based Web site as 
a learning environment for teachers. We learned that applying checklist 
criteria can provide an efficient means for gathering formative feedback 
during the site development process and can be useful for comparing the 
content of our site with an established framework. Ffowever, checklist 
criteria can also be subject to the reviewer’s assumptions and preferences. 
Data collection methods, such as Web site traffic analysis software, can 
be cost-effective and initially labor saving because they can be employed 
remotely or automatically. Statistical data can help teacher educators who 
are designing Web sites learn how the intended audience uses the online, 
learning environment, how to enhance its appeal, and how to enlarge the 
user base by parmering with referrer sites. The complementary think aloud 
protocol, however, can be used to reveal an even more complete picture of 
what engages users and encourages them to return for additional learning. 
The think aloud protocol can reveal how easy or difficult and how useful 
the intended audience finds the typical tasks within the learning environ- 
ment. While this feedback can be insightful, one must apply the protocol 
in ways that capture participant’s thinking aloud, and consider the influ- 
ence of their previous skills and experiences. In addition, one must ensure 
that design tasks are focused and appropriate in their level of difficulty 
and are representative of actual tasks users are intended to undertake at 
the Web site. Surveys prior to using the protocol and interviews following 
its application could help to provide this information. 

Online, resource-based learning environments hold tremendous 
potential for learning in both traditional and nontraditional settings. As 
networked technologies become increasingly ubiquitous and technology 
capabilities increasingly sophisticated, organizations and institutions 
serving preservice and inservice teachers with an educational mission are 
turning to the Web to provide more affordable, customizable outreach and 
resources for instruction. Further investigation is needed to develop an 
established, multi-method approach to gathering formative feedback on 
the potential and effectiveness of such resource-based Web sites as learning 
environments. Sustained and concomitant application of data gathering 


methods would do much to advance the Web site development process 
because it would help designers refine the environment in ways that more 
closely align with the ongoing behaviors, expressed needs, and interests 
of its intended users. Systematic, comprehensive evaluation, however, is 
usually difficult to accomplish during the initial design, prototyping, and 
beta-testing phases of Web site development, especially for organizations 
and public sector institutions with limited resources on time-sensitive 
grant schedules, or by individuals or groups of volunteers. Automated 
data gathering software, that can be embedded “behind the scenes” in the 
learning environment, holds promise for real time feedback, especially 
where it is part of a comprehensive, predetermined plan for data collection 
and analysis. Further development of electronic tools and instruments 
for the evaluation of online, resource-based learning environments and 
additional examples of their application would help teacher educators as 
designers meet their goals of providing quality learning opportunities for 
the intended learners anytime and anywhere. 
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Appendix: 

Talk Aloud /Thinking Aloud Protocol 

Main instructions 

“I’m going to ask you to do several short tasks involving looking through 
a couple Weh sites. The basic idea in the talk aloud exercise is to have you 
describe everything that you’re doing, seeing, and thinking while you go 
through each task. It’s more than just talking aloud to yourself as you 
might do in other situations, since you’ll be reporting all your thoughts 
while going through this exercise. But, it’s not the same thing as being a 
teacher, either, trying to explain the reasoning behind everything you do 
with the Web site. Just think, reason in a loud voice, tell me everything 
that passes through your head while you work through the task. There is 
no right or wrong way to go through this task.” 

Reminders 

After 20 seconds of silence, the experimenter should say, “What are you 
thinking about?” 

Tasks 

Practice Task: From the university’s homepage, find the current hours for 
[name of the main] library. 


Task #1: Use the eTech Web site to find one new idea for integrating 
technology into a science lesson for fourth grade students. 

Task #2: Use the eTech Web site to find a simple guide to decide whether 
technology is appropriate to use in a lesson. 

Task #3: Use the eTech Web site to find one new idea for integrating 
technology into a geometry lesson for eleventh grade students. 

Task #4: Use the eTech Web site to find one resource to aid in teaching 
other preservice teachers about technology integration. 

interview Questions 

Is there any topic listed on the Web site that you didn’t explore but would 
have liked to look at? 

Did you find it easy or difficult to navigate around the Web site? What 
made it easy or difficult? 

What do you think is most useful about this Web site as a resource for 
teachers? 

What do you think is least useful about this Web site as a resource for 
teachers? 

What improvements, if any, can you see that would make the Web site 
a better resource for teachers? 


Editor’s Remarks continued from p. 2 

accompanying blog will also be made available so that you can participate 
by contributing your thoughts. 

Th is Iss u e 

The four articles in this issue provide further rationale for the need for 
advocacy discussed in this column. Each of these articles suggests some 
of the exciting affordances of technolog)'^ to enhance both teacher educa - 
tion and student learning. 

In “A Comparison of Teacher Education Faculy and Preservice 
Teacher Technolog)^ Competence” Jim Carroll and Patricia Morrell share 
results from a study comparing teacher education faculty members and 
preservice teachers with respect to self - perceptions of technolog)^ com - 
petence. Findings from this study point to both faculy members and 
students as having technological expertise and that each group might 
benefit from the capabilit)' of the other. The study presents a positive 
picture of teacher education faculy members and their abilities to develop 
student knowledge of meaningful technolog)' applications. 

The work of Beverly Ray and Martha Hocutt reported in “Teacher - 
createdi Teacher - centered Weblogs: Perceptions and Practice s ” provides 
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useful insights for teacher educators interested in exploring the possibili - 
ties of Weblogs for classroom teachers. This qualitative study focuses 
upon determining possible themes that emerge as teachers create and 
use Weblogs to enhance their professional practice. 

Teacher educators and classroom teachers working to design online 
environments for teacher professional development will find the work of 
Christine Greenhow, Sara Dexter and Eric Riedel useful as they provide 
three useful tools for evaluating both the pedagogical design and user - 
centered functionality of these sites. 

Using technolog)'^ to enhance opportunities for teacher reflection is 
also the major theme for Lynn Bryan and Art Recesso in their study of 
the use of a video analysis tool to help enhance science teacher reflective 
practice. In “ Promoting Reflection among Science Student Teachers us - 
ing a Web based Video Analysis Tool,” the authors explore a user friendly 
video analysis tool in helping both preservice and inservice teachers 
engage in more reflective practice. 

jiVll four of the articles in this issue represent innovative approaches 
to improving our use of technolog)^ in teacher education and provide a 
glimpse of some of the possibilities for future work. The urgent need for 
legislative advocacy suggested in this column is further reinforced by the 
forward - looking projects highlighted in this issue. 


President's Message continued from p. 3 

posals. We’ll post further guidelines for possible SIGTE financial support 
to graduate student presenters at NECC 2007, as a way of encouraging 
future teacher educators to get involved in SIGTE activities. 

Although recognition of our 2005 - 2006 SIGTE volunteers took place 
at the SIGTE business meeting, I want to repeat my thanks to Melissa 
Pierson for her outstanding service as President from 200 4- 2006; her 
gracious and enthusiastic leadership increased active participation of 
SIGTE members. Thanks also to Karen Grove who was elected to a 
second term as Treasurer. Finally, at the Business Meeting, we recognized 
deceased member Bill Halverson’s many fine contributions to SIGTE 
and the educational communit)'. As a community, we have much to be 
proud of and thankful for in the way of member contributions to the 
profession, both past and present . 
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